Skip to content

Fall back to remote runtime on Spark Connect when the legacy namespace is unavailable#1464

Closed
sd-db wants to merge 2 commits into
databricks:mainfrom
sd-db:fix/runtime-spark-connect-import
Closed

Fall back to remote runtime on Spark Connect when the legacy namespace is unavailable#1464
sd-db wants to merge 2 commits into
databricks:mainfrom
sd-db:fix/runtime-spark-connect-import

Conversation

@sd-db

@sd-db sd-db commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Importing databricks.sdk.runtime on a Spark Connect runtime no longer raises CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT. When the legacy user namespace can't be materialized, the import now logs a warning and falls back to the existing Spark Connect-compatible remote implementation. This fixes the upstream crash behind dbt-databricks#1252.

Fixes #1463

Why

WorkspaceClient.__init__ eagerly builds dbutils via _make_dbutils, which on a cluster imports databricks.sdk.runtime. That import calls UserNamespaceInitializer.getOrCreate().get_namespace_globals(), materializing a legacy SparkContext. On a shared-access-mode (Spark Connect) cluster this raises CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT — a pyspark.errors.PySparkRuntimeError, not an ImportError — so the existing except ImportError: doesn't catch it and the error escapes the import, crashing WorkspaceClient construction before any API call.

Because the failure is in the constructor, consumers can't guard it with a try/except around ws.dbutils, and there's no config/env switch to skip the eager build — so downstream tools resort to monkey-patching the private _make_dbutils. The except ImportError branch is already the Spark Connect-compatible path (it builds spark via DatabricksSession and dbutils via RemoteDbUtils), so this PR routes the materialization failure there.

See #1463 for the full analysis. A complementary follow-up (making WorkspaceClient.dbutils lazy) is noted there as a separate discussion since it touches generated code.

What changed

Interface changes

None.

Behavioral changes

On a Spark Connect runtime, importing databricks.sdk.runtime now logs a WARNING and uses the remote implementation instead of raising at import time. When dbruntime is absent, or when the namespace materializes successfully, behavior is unchanged.

Internal changes

Restructured the runtime-namespace block in databricks/sdk/runtime/__init__.py into a single try with sibling except ImportError (not a classic runtime) and except Exception (Spark Connect / CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT, logged) clauses, plus an if not _use_runtime_namespace: guard over the existing — unchanged — OSS/remote block.

How is this tested?

New tests/test_runtime.py simulates a Spark Connect runtime by injecting a fake dbruntime whose get_namespace_globals() raises CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT, then asserts that (a) reloading databricks.sdk.runtime survives and falls back (is_local_implementation is True, dbutils is not None) and (b) WorkspaceClient(config=…) constructs without raising — the direct reproduction of the reported failure. Verified red→green (both tests fail with the change reverted, pass with it). The full unit suite shows no regressions introduced by this change.

…e is unavailable

On a Databricks shared-access-mode (Spark Connect) cluster, importing databricks.sdk.runtime (which happens when WorkspaceClient.__init__ eagerly builds dbutils) materializes a legacy SparkContext via UserNamespaceInitializer.get_namespace_globals() and raises CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT (a PySparkRuntimeError, not ImportError). The surrounding 'except ImportError' does not catch it, so the error escapes the import and crashes WorkspaceClient construction.

Treat a namespace-materialization failure the same as 'not in a classic runtime': log a warning and fall back to the existing OSS/remote implementation, which is Spark Connect-compatible (DatabricksSession + RemoteDbUtils).

Fixes databricks#1463

Signed-off-by: Shubham Dhal <shubham.dhal@databricks.com>
@sd-db sd-db temporarily deployed to test-trigger-is June 8, 2026 08:39 — with GitHub Actions Inactive
@Divyansh-db Divyansh-db temporarily deployed to test-trigger-is June 9, 2026 14:55 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/sdk-py

Inputs:

  • PR number: 1464
  • Commit SHA: 337cc3868bebddf9273e638dc945ea180957736c

Checks will be approved automatically on success.

@Divyansh-db

Copy link
Copy Markdown
Contributor

Hi @sd-db — thanks for the detailed analysis and the fix. Carrying this forward upstream.

Tests don't run on forked PRs, so the cleanest move was to land the same commit on an upstream branch. Opened #1469 with your commit preserved.

Closing this in favor of #1469. Thanks again for the contribution!

@Divyansh-db Divyansh-db closed this Jun 9, 2026
pull Bot pushed a commit to Future-Outlier/databricks-sdk-py that referenced this pull request Jun 11, 2026
…e is unavailable (databricks#1469)

## Summary

Importing `databricks.sdk.runtime` on a Spark Connect runtime (e.g.
shared-access-mode clusters) no longer raises
`CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT` at import time. When the legacy
user namespace cannot be materialized, the import now logs a warning and
falls back to the existing Spark Connect-compatible remote
implementation, so `WorkspaceClient()` construction succeeds on such
clusters.

Fixes databricks#1463. Carries forward @sd-db's work from databricks#1464 (closed because
fork PRs in this repo cannot run tests).

## Why

`WorkspaceClient.__init__` eagerly builds `dbutils` via `_make_dbutils`,
which on a cluster does `from databricks.sdk.runtime import dbutils`.
That import calls
`UserNamespaceInitializer.getOrCreate().get_namespace_globals()`,
materializing a legacy `SparkContext`. On a Spark Connect cluster this
raises `CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT` — a
`pyspark.errors.PySparkRuntimeError`, not an `ImportError` — so the
existing `except ImportError:` does not catch it and the error escapes
the import, crashing `WorkspaceClient` construction before any API call.
This is what databricks/dbt-databricks#1252 hits in Python models on
shared clusters.

The existing `except ImportError` branch is already the Spark
Connect-compatible path (it builds `spark` via `DatabricksSession` and
`dbutils` via `RemoteDbUtils`), so this PR routes the materialization
failure there.

A complementary follow-up — making `WorkspaceClient.dbutils` lazy via a
`cached_property` so consumers that never touch it skip the build
entirely — is noted in databricks#1463 as a separate discussion since it touches
generated code. Related issue databricks#986 (off-cluster eager `RemoteDbUtils`
auth failure) is the symmetric case and is intentionally not addressed
here; the lazy-dbutils follow-up would unify both.

## What changed

### Behavioral changes

On a Spark Connect runtime, importing `databricks.sdk.runtime` now logs
a `WARNING` and uses the remote implementation instead of raising at
import time. When `dbruntime` is absent (off-cluster) or the namespace
materializes successfully (classic runtime), behavior is unchanged.

### Internal changes

`databricks/sdk/runtime/__init__.py`: the runtime-namespace block is
restructured into a single `try` with sibling `except ImportError`
(existing — "not in a classic runtime") and `except Exception` (new —
Spark Connect / `CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT`, logged)
clauses, plus an `if not _use_runtime_namespace:` guard over the
existing — unchanged — OSS/remote block. The catch is intentionally
broad rather than typed on `PySparkRuntimeError` to avoid pulling
`pyspark` in at SDK import time just to narrow the exception type; the
inline comment notes this.

## How is this tested?

New `tests/test_runtime.py` simulates a Spark Connect runtime by
injecting a fake `dbruntime` whose `get_namespace_globals()` raises
`CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT`, and asserts that:
- reloading `databricks.sdk.runtime` survives the failure and falls back
(`is_local_implementation is True`, `dbutils is not None`)
- `WorkspaceClient(config=…)` constructs without raising — the direct
reproduction of the reported failure

Verified red→green locally. Full unit test suite (2098 tests) passes
with no regressions.

---------

Signed-off-by: Shubham Dhal <shubham.dhal@databricks.com>
Signed-off-by: Divyansh Vijayvergia <171924202+Divyansh-db@users.noreply.github.com>
Co-authored-by: Shubham Dhal <shubham.dhal@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WorkspaceClient construction fails on Spark Connect clusters: CONTEXT_UNAVAILABLE_FOR_REMOTE_CLIENT when importing databricks.sdk.runtime

2 participants